Automated procedure evaluation

ABSTRACT

A robotic system is configured to evaluate an identified phase of a medical procedure. The robotic system includes a video capture device; a robotic manipulator; one or more sensors; an input device; a data store; and control circuitry. The control circuitry is configured to: determine a first status of the robotic manipulator based on sensor data from the one or more sensors; identify a first input from the input device for initiating a first action of the robotic manipulator; perform a first analysis of a video of a patient site captured by the video capture device; identify a first phase of the medical procedure based at least in part on the first status of the robotic manipulator, the first input, and the first analysis of the video; and generate an evaluation of the first phase of the medical procedure based on one or more metrics associated with the first phase.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of International Application No.PCT/IB2021/060596, filed on Nov. 16, 2021 and entitled AUTOMATEDPROCEDURE EVALUATION, which claims the benefit of priority to U.S.Provisional Application No. 63/116,798, filed Nov. 20, 2020, and U.S.Application No. 63/132,875, filed Dec. 31, 2020, each of which is herebyincorporated by reference in its entirety.

BACKGROUND Field

The present disclosure relates to the field of medical devices andprocedures and artificial intelligence assisted data processing.

Description of the Related Art

Various medical procedures involve the use of a robotic system thatassists with using one or more medical instruments configured topenetrate the human anatomy to reach a treatment site. Certainoperational processes can involve inserting the one or more medicalinstruments through the skin or an orifice of a patient to reach thetreatment site and extract an object from the patient, such as a urinarystone.

SUMMARY

Described herein are one or more systems, devices, and/or methods toassist a physician or other medical professional in controlling amedical instrument to access an object, such as a urinary stone, locatedwithin the human anatomy.

For purposes of summarizing the disclosure, certain aspects, advantages,and novel features have been described. It is to be understood that notnecessarily all such advantages may be achieved in accordance with anyparticular embodiment. Thus, the disclosed embodiments may be carriedout in a manner that achieves or optimizes one advantage or group ofadvantages as taught herein without necessarily achieving otheradvantages as may be taught or suggested herein.

A system of one or more computers can be configured to performparticular operations or actions by virtue of having software, firmware,hardware, or a combination of them installed on the system that inoperation causes or cause the system to perform the actions. One or morecomputer programs can be configured to perform particular operations oractions by virtue of including instructions that, when executed by dataprocessing apparatus, cause the apparatus to perform the actions. Onegeneral aspect includes a robotic system for evaluating an identifiedphase of a medical procedure performed by the robotic system. Therobotic system also includes a video capture device; a roboticmanipulator; one or more sensors configured to detect a configuration ofthe robotic manipulator; an input device configured to receive one ormore user interactions and initiate one or more actions by the roboticmanipulator; a data store configured to store metrics associated withphases of medical procedures; and control circuitry communicativelycoupled to the input device and robotic manipulator. The controlcircuitry is configured to: determine a first status of the roboticmanipulator based on sensor data from the one or more sensors; identifya first input from the input device for initiating a first action of therobotic manipulator; perform a first analysis of a video of a patientsite captured by the video capture device; identify a first phase of themedical procedure based at least in part on the first status of therobotic manipulator, the first input, and the first analysis of thevideo; and generate an evaluation of the first phase of the medicalprocedure based on one or more metrics associated with the first phase.Other embodiments of this aspect include corresponding computer systems,apparatus, and computer programs recorded on one or more computerstorage devices, each configured to perform the actions of the methods.

Implementations may include one or more of the following features. Thefirst phase of the medical procedure may include one of ureteroscopydriving, ureteroscopy lasing, ureteroscopy basketing, and percutaneousneedle insertion. The first phase may include ureteroscopy basketing andgenerating the evaluation may include: counting a number of basketoperations; counting a number of ureteroscope retractions; determining aratio of the number of basket operations to the number of ureteroscoperetractions; and comparing the determined ratio with other ratios fromprevious ureteroscopy basketing procedures. The first phase may includeureteroscopy driving and generating the evaluation may include: countinga number of times a user drives a scope manually; counting a number oftimes the user drives the scope robotically; determining a ratio of thenumber of times the user drives a scope manually to the number of timesthe user drives the scope robotically; and comparing the determinedratio with other ratios from previous ureteroscopy basketing procedures.The first phase may include percutaneous needle insertion and generatingthe evaluation may include: counting a number of times a user attemptsto insert a needle until the user successfully inserts the needle; andcomparing the counted number of times with recorded needle insertionattempts from previous percutaneous needle insertion operations. Thefirst phase may include percutaneous needle insertion and generating theevaluation may include: counting time taken to survey a kidney beforeselecting a target calyx for percutaneous access; and comparing thecounted time with recorded times from previous percutaneous needleinsertion operations. The first phase may include percutaneous needleinsertion and generating the evaluation may include: counting a numberof times a navigational field generator for tracking a needle isrepositioned; and comparing the counted number of times with recordedrepositioning numbers from previous percutaneous needle insertionoperations. The first phase may include percutaneous needle insertionand generating the evaluation may include: counting a number of times anautomated alignment of an end effector of the robotic manipulator with acatheter is initiated; and comparing the counted number of times withrecorded automated alignment numbers from previous operations. The firstphase may include ureteroscopy lasing and generating the evaluation mayinclude: counting a lasing time for a stone; determining a size of thestone; and comparing a ratio of the lasing time to the size of the stonewith previous ratios from other operations. The first phase may includeureteroscopy lasing and generating the evaluation may include:determining a type of the stone; and aggregating statistics acrosssurgical operations based on the type of the stone. The first phase mayinclude ureteroscopy lasing and generating the evaluation may include:counting a number of times a view of the video capture device becomesoccluded by dust from fragmentation of a stone; and comparing thecounted number of times with recorded number of dust occlusions fromoperations. Implementations of the described techniques may includehardware, a method or process, or computer software on acomputer-accessible medium.

One general aspect includes a method for evaluating an identified phaseof a medical procedure performed by a robotic system that may include avideo capture device. The method also includes determining a firststatus of the robotic manipulator based on sensor data from the one ormore sensors; identifying a first input from the input device forinitiating a first action of the robotic manipulator; performing a firstanalysis of a video of a patient site captured by the video capturedevice; identifying a first phase of the medical procedure based atleast in part on the first status of the robotic manipulator, the firstinput, and the first analysis of the video; and generating an evaluationof the first phase of the medical procedure based on one or more metricsassociated with the first phase. Other embodiments of this aspectinclude corresponding computer systems, apparatus, and computer programsrecorded on one or more computer storage devices, each configured toperform the actions of the methods.

Implementations may include one or more of the following features. Themethod where the first phase may include ureteroscopy basketing andgenerating the evaluation may include: counting a number of basketoperations; counting a number of ureteroscope retractions; determining aratio of the number of basket operations to the number of ureteroscoperetractions; and comparing the determined ratio with other ratios fromprevious ureteroscopy basketing operations. The first phase may includeureteroscopy driving and generating the evaluation may include: countinga number of times a user drives a scope manually; counting a number oftimes the user drives the scope robotically; determining a ratio of thenumber of times the user drives a scope manually to the number of timesthe user drives the scope robotically; and comparing the determinedratio with other ratios from previous ureteroscopy basketing operations.The first phase may include percutaneous needle insertion and generatingthe evaluation may include: counting a number of times a user attemptsto insert a needle until the user successfully inserts the needle; andcomparing the counted number of times with recorded needle insertionattempts from previous percutaneous needle insertion operations. Thefirst phase may include percutaneous needle insertion and generating theevaluation may include: counting time taken to survey a kidney beforeselecting a target calyx for percutaneous access; and comparing thecounted time with recorded times from previous percutaneous needleinsertion operations. The first phase may include percutaneous needleinsertion and generating the evaluation may include: counting a numberof times a navigational field generator for tracking a needle isrepositioned; and comparing the counted number of times with recordedrepositioning numbers from previous percutaneous needle insertionoperations. The first phase may include percutaneous needle insertionand generating the evaluation may include: counting a number of times anautomated alignment of an end effector of the robotic manipulator with acatheter is initiated; and comparing the counted number of times withrecorded automated alignment numbers from previous operations. The firstphase may include percutaneous antegrade ureteroscopy lasing andgenerating the evaluation may include: counting a lasing time for astone; determining a size of the stone; and comparing a ratio of thelasing time to the size of the stone with previous ratios from otheroperations. The first phase may include ureteroscopy lasing andgenerating the evaluation may include: determining a type of the stone;and aggregating statistics across surgical operations based on the typeof the stone. The first phase may include ureteroscopy lasing andgenerating the evaluation may include: counting a duration during whicha view of the video capture device becomes occluded by dust fromfragmentation of a stone; and comparing the counted duration withrecorded durations from previous operations. Implementations of thedescribed techniques may include hardware, a method or process, orcomputer software on a computer-accessible medium.

One general aspect includes a control system of a robotic device forevaluating an identified phase of a medical procedure. The controlsystem also includes a communication interface configured to receivesensor data, user input data, and video data from the robotic device;memory configured to store the sensor data, the user input data, and thevideo data; and one or more processors configured to: determine a firststatus of a manipulator of the robotic device based on sensor data fromthe one or more sensors; identify a first input, from the user inputdata, for initiating a first action of the manipulator; perform a firstanalysis of a video of a patient site captured by the video capturedevice; identify a first phase of the medical procedure based at leastin part on the first status of the manipulator, the first input, and thefirst analysis of the video. The system can also generate an evaluationof the first phase of the medical procedure based on one or more metricsassociated with the first phase. Other embodiments of this aspectinclude corresponding computer systems, apparatus, and computer programsrecorded on one or more computer storage devices, each configured toperform the actions of the methods.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments are depicted in the accompanying drawings forillustrative purposes and should in no way be interpreted as limitingthe scope of the disclosure. In addition, various features of differentdisclosed embodiments can be combined to form additional embodiments,which are part of this disclosure. Throughout the drawings, referencenumbers may be reused to indicate correspondence between referenceelements.

FIG. 1 illustrates an example medical system to perform or assist inperforming medical procedures, according to certain embodiments.

FIGS. 2A-2B illustrate a perspective view of the medical system whileperforming a urinary stone capture procedure, according to certainembodiments.

FIG. 3 illustrates a block diagram of a control system of the medicalsystem, with associated inputs and outputs, according to certainembodiments.

FIG. 4A illustrates a block diagram of a control system configured toutilize machine learning to generate outputs from video data, accordingto certain embodiments.

FIG. 4B illustrates a block diagram of a control system configured toutilize machine learning to generate outputs from several types of data,according to certain embodiments.

FIG. 5 is a flow diagram of a phase identification process, according tocertain embodiments.

FIG. 6 is a flow diagram of a triggering process for automated roboticaction, according to certain embodiments.

FIG. 7 is a diagram showing different types of triggered actions of therobotic system, according to certain embodiments.

FIG. 8 is a flow diagram of an evaluation process for tasks performedduring identified phases, according to certain embodiments.

FIG. 9 is a flow diagram of a scoring process for medical tasks,according to certain embodiments.

FIG. 10 is a flow diagram of another scoring process for medical tasks,according to certain embodiments.

FIG. 11 illustrates example details of the robotic system, according tocertain embodiments.

FIG. 12 illustrates example details of the control system, according tocertain embodiments.

DETAILED DESCRIPTION

The headings provided herein are for convenience only and do notnecessarily affect the scope or meaning of disclosure. Although certainpreferred embodiments and examples are disclosed below, the subjectmatter extends beyond the specifically disclosed embodiments to otheralternative embodiments and/or uses and to modifications and equivalentsthereof. Thus, the scope of the claims that may arise herefrom is notlimited by any of the particular embodiments described below. Forexample, in any method or process disclosed herein, the acts oroperations of the method or process may be performed in any suitablesequence and are not necessarily limited to any particular disclosedsequence. Various operations may be described as multiple discreteoperations in turn, in a manner that may be helpful in understandingcertain embodiments; however, the order of description should not beconstrued to imply that these operations are order dependent.Additionally, the structures, systems, and/or devices described hereinmay be embodied as integrated components or as separate components. Forpurposes of comparing various embodiments, certain aspects andadvantages of these embodiments are described. Not necessarily all suchaspects or advantages are achieved by any particular embodiment. Thus,for example, various embodiments may be carried out in a manner thatachieves or optimizes one advantage or group of advantages as taughtherein without necessarily achieving other aspects or advantages as mayalso be taught or suggested herein.

Certain standard anatomical terms of location may be used herein torefer to the anatomy of animals, and namely humans, with respect to thepreferred embodiments. Although certain spatially relative terms, suchas “outer,” “inner,” “upper,” “lower,” “below,” “above,” “vertical,”“horizontal,” “top,” “bottom,” and similar terms, are used herein todescribe a spatial relationship of one device/element or anatomicalstructure to another device/element or anatomical structure, it isunderstood that these terms are used herein for ease of description todescribe the positional relationship between element(s)/structures(s),as illustrated in the drawings. It should be understood that spatiallyrelative terms are intended to encompass different orientations of theelement(s)/structures(s), in use or operation, in addition to theorientations depicted in the drawings. For example, an element/structuredescribed as “above” another element/structure may represent a positionthat is below or beside such other element/structure with respect toalternate orientations of the subject patient or element/structure, andvice-versa.

Overview

The present disclosure relates to techniques and systems for collectingand analyzing data from robotic assisted medical procedures, such asthose performed by a robotic system for stone management (e.g.,retrieving urinary stones, suction of stone fragments, etc.) orperforming other medical procedures. A medical procedure may progressthrough several different phases. For example, in ureteroscopy, phasescan include percutaneous insertion of a medical instrument into thebody, travel to the urinary stone location, lasing of the urinary stone,and/or basketing of the broken up stone. Robotic systems typically haveseveral sensors and input devices, allowing the generation of a largeamount of data during the medical procedure. This procedure data can beused to automatically determine the different phases of the operation.By identifying these phases, the robotic system can anticipate andprepare for the actions of the medical professional operating therobotic system during the medical procedure.

A medical system comprising the robotic system may also allow annotatingof video footage of the procedure with metadata identifying thedifferent phases. This allows the video footage to be more easilyreviewed by users as well as allowing for more sophisticated analysis ofthe video footage using artificial intelligence (AI). This can make iteasier to evaluate and score the actions performed by the user oroperator by comparing those actions to similar actions fromcorresponding phases performed during other procedures. For example, thevideo footage and associated data can be analyzed by AI systems togenerate statistics for the operation, such as attempts before successper phase or entire procedure, time of each phase, number ofarticulation commands provided by the operator, accuracy of the needleinsertion, and/or the like. Furthermore, data can be aggregated overseveral operations and used to generate statistics for the type ofoperation in general, such as success rates, average operation times perphase or entire procedures, and/or the like. Such a medical system canalso provide additional benefits, such as by generating case summaries.

In one example scenario, there are distinct phases during percutaneousrenal access or other procedures. In an exemplary workflow, the userdrives the scope to a desired calyx, marks the papilla and retracts thescope to see the target papilla. The user then holds the needle, selectsan insertion site, and aligns the needle trajectory with the targetpapilla using a graphical user interface (“GUI”). Finally, the userinserts the needle while following the graphical user interface to gainaccess to the kidney through the target papilla. To improve procedureefficiency and assess user skill, the medical system can label thebeginning and end of these events and obtain ground truth data onwhether the percutaneous access (“perc”) attempt was successful or not.

After dividing the case data into distinct phases and generating a phasetransition chart showing these phases, the transition chart can be usedto evaluate the procedure. For example, one example transition chart mayshow that the physician selected a target and an insertion site, but didnot move forward with needle alignment step, and instead drove to adifferent calyx to select a new target. The chart may show that thephysician did not get a visual confirmation of access in the firstpercutaneous access attempt and drove the scope to locate the needle.The chart may show that the physician did another percutaneous accessattempt with the same target, and this time gained visual confirmation.Such a chart can be displayed on a GUI of the medical system, as adigital or print report, on a mobile application, and/or similar type ofoutput.

Another potential benefit is providing ground truth (success/fail)annotation. Phase segmentation can enable predicting if a certainpercutaneous access attempt was successful or not, thereby serving as aground truth for the case. The medical system can track a set of featuredescriptors during the needle insertion phase to determine ifpercutaneous access has been successful. A feature descriptor caninclude various quantities or metrics measured by the medical system,such as needle and scope velocity, and relative pose of the needle withrespect to the scope. It can also include scope articulation commandsand features detected by a computer vision algorithm which detects ifthe needle is visible in camera view and quantifies how much anatomicalmotion there is. For example, there can be a direct correlation betweenvisual confirmation and success. In one scenario, if the computer visionalgorithm detects a needle in the endoscopic view, the percutaneousaccess attempt can be annotated or otherwise indicated as successful. Inanother scenario, the distance between needle and scope may be verysmall, but there is no visual confirmation of the needle on the scope.If the scope starts moving, it implies that the percutaneous accessattempt was unsuccessful and the user is looking for the needle ordriving to another calyx to select a new target. Thus, detection of thescope movement in that situation can be used to annotate or otherwiseindicate that the percutaneous access attempt as unsuccessful.

Another potential benefit is providing skill assessment. Phasesegmentation can enable running phase specific data analysis to assessphysician skill and compute case statistics intraoperatively orpostoperatively. The table below shows postoperative metrics for some ofthe percutaneous access phases. For example, by knowing when needleinsertion starts (e.g., identified via video capture, sensor data, orthe like), the medical system can determine the entry point on the skin(e.g., using kinematic data, video analysis or the like) and computesite selection metrics such as tract length (e.g., distance from skin topapilla).

SCOPE DRIVING PHASE NEEDLE INSERTION PHASE SITE SELECTION PHASE Numberof Articulation Needle Insertion Site Selection Average Tract CommandsSuccess Rate Accuracy Time Length This All Lower Mid Upper Expert AllExpert All Lower Mid Upper case cases Pole Pole Pole Average casesAverage cases Pole Pole Pole

For example, in the scope driving phase, the user's skill can beevaluated based on the number of articulation commands received by thesystem. If less commands are received, that implies that the operationhas gone smoothly, indicating greater skill. If more commands arereceived, that implies that multiple attempts have to be performed,indicating room for improvement. These metrics can also provideinformation on which parts of the anatomy a user struggles navigating.The number of articulations commands may be recorded and/or displayedfor this operation or for multiple operations (all cases, all cases in acertain period, all cases performed by the user, etc.). For example, themedical system can generate metrics that compare by location for a givencase, across physicians, and/or for the same physician over time overmultiple operations.

In another example, in the needle insertion phase, the user's skill canbe evaluated based on the success rate and/or the needle insertionaccuracy. Success rate can be further specifically calculated based onkidney location, such as at the lower pole, mid pole, or upper pole.Needle insertion accuracy can be compared to an average value forexperts. The needle insertion accuracy may be recorded and/or displayedfor this operation or for multiple operations (e.g., all cases, allcases in a certain period, all cases performed by the user, etc.).

In a further example, in the site selection phase, the user's skill canbe evaluated based on the site selection time or time taken by the userto select a site and average tract length. The site selection time canbe compared to an average value for experts. The site selection time maybe recorded and/or displayed for this operation or for multipleoperations (all cases, all cases in a certain period, all casesperformed by the user, etc.). Average track length can be furtherspecifically calculated based on kidney location, such as at the lowerpole, mid pole, or upper pole. Tract length for a patient may be used asan indicator of patient body mass index (BMI). This can allow caseperformance to be aggregated based on patient population characteristicssuch as BMI values or ranges.

The above table shows just some examples of possible metrics than can beevaluated. Furthermore, the above table shows just some of thespecificity that can be applied to those metrics. For example, some ofthe specificity applied for one metric can be applied to other metrics.In some embodiments, needle insertion accuracy can be broken downfurther based on kidney location. Success rate can be shown with morespecificity by comparing to an expert average or across multipleoperations (e.g., all cases, all cases in a certain period, all casesperformed by the user, etc.).

Another potential benefit of such a medical system is providing skillassessment workflow optimization. Workflow analysis can show thecorrelation between the sequence of workflow steps and percutaneousaccess success and efficiency. For example, the algorithm can comparecases where site selection was performed before target selection versustarget selection performed before site selection and assess the impactto percutaneous access time and accuracy.

Such a medical system can be used in several types of procedures,including ureteroscopy. Kidney stone disease, also known asurolithiasis, is a relatively common medical condition that involves theformation, in the urinary tract, of a solid piece of material, referredto as “kidney stones,” “urinary stones,” “renal calculi,” “renallithiasis,” or “nephrolithiasis.” Urinary stones can be formed and/orfound in the kidneys, the ureters, and the bladder (referred to as“bladder stones”). Such urinary stones form as a result of concentratedminerals and can cause significant abdominal pain once they reach a sizesufficient to impede urine flow through the ureter or urethra. Urinarystones can be formed from calcium, magnesium, ammonia, uric acid,cysteine, and/or other compounds.

To remove urinary stones from the bladder and ureter, surgeons caninsert a ureteroscope into the urinary tract through the urethra.Typically, a ureteroscope includes an endoscope at its distal endconfigured to enable visualization of the urinary tract. Theureteroscope can also include a lithotomy mechanism, such as the basketretrieval device, to capture or break apart urinary stones. During aureteroscopy procedure, one physician/technician can control theposition of the ureteroscope, while another other physician/techniciancan control the lithotomy mechanism.

In many embodiments, the techniques and systems are discussed in thecontext of a minimally invasive procedure. However, it should beunderstood that the techniques and systems can be implemented in thecontext of any medical procedure including, for example, percutaneousoperations where access is gained to a target location by making apuncture and/or a minor incision into the body to insert a medicalinstrument, non-invasive procedures, therapeutic procedures, diagnosticprocedures, non-percutaneous procedures, or other types of procedures.For example, such techniques can be used in tumor biopsy or ablation forurology and bronchoscopy, where an automated biopsy operation can betriggered when the system detects proximity to a suspicious site. Anendoscopic procedure can include a bronchoscopy, a ureteroscopy, agastroscopy, nephroscopy, nephrolithotomy, and so on. Further, in manyembodiments, the techniques and systems are discussed as beingimplemented as robotically-assisted procedures. However, it should alsobe appreciated that the techniques and systems can be implemented inother procedures, such as in fully-robotic medical procedures.

For ease of illustration and discussion, the techniques and systems arediscussed in the context of removing urinary stones, such as kidneysstones from the kidneys. However, as noted above, the techniques andsystems can be used to perform other procedures.

Medical System

FIG. 1 illustrates an example medical system 100 to perform or assist inperforming medical procedures in accordance with one or moreembodiments. Embodiments of the medical system 100 can be used forsurgical and/or diagnostic procedures. The medical system 100 includes arobotic system 110 configured to engage with and/or control a medicalinstrument 120 to perform a procedure on a patient 130. The medicalsystem 100 also includes a control system 140 configured to interfacewith the robotic system 110, provide information regarding theprocedure, and/or perform a variety of other operations. For example,the control system 140 can include a display 142 to present a userinterface 144 to assist the physician 160 in using the medicalinstrument 120. Further, the medical system 100 can include a table 150configured to hold the patient 130 and/or an imaging sensor 180, such asa camera, x-ray, computed tomography (CT), magnetic resonance imaging(MRI), positron emission tomography (PET) device, or the like.

In some embodiments, the physician performs a minimally-invasive medicalprocedure, such as ureteroscopy. The physician 160 can interact with thecontrol system 140 to control the robotic system 110 to navigate themedical instrument 120 (e.g., a basket retrieval device and/or scope)from the urethra up to the kidney 170 where the stone 165 is located.The control system 140 can provide information via a display 142regarding the medical instrument 120 to assist the physician 160 innavigation, such as real-time images from the medical instrument 120 orthe imaging sensor 180. Once at the site of the kidney stone, themedical instrument 120 can be used to break-up and/or capture a urinarystone 165.

In some implementations of using the medical system 100, a physician 160can perform a percutaneous procedure. To illustrate, if the patient 130has a kidney stone 165 in a kidney 170 that is too large to be removedthrough a urinary tract, the physician 160 can perform a procedure toremove the kidney stone through a percutaneous access point on thepatient 130. For example, the physician 160 can interact with thecontrol system 140 to control the robotic system 110 to navigate themedical instrument 120 (e.g., a scope) from the urethra up to the kidney170 where the stone 165 is located. The control system 140 can provideinformation via a display 142 regarding the medical instrument 120 toassist the physician 160 in navigating the medical instrument 120, suchas real-time images from the medical instrument 120 or the imagingsensor 180. Once at the site of the kidney stone, the medical instrument120 can be used to designate a target location for a second medicalinstrument (not shown) to access the kidney percutaneously (e.g., adesired point to access the kidney). To minimize damage to the kidney,the physician 160 can designate a particular papilla as the targetlocation for entering into the kidney with the second medicalinstrument. However, other target locations can be designated ordetermined. Once the second medical instrument has reached the targetlocation, the physician 160 can use the second medical instrument and/oranother medical instrument to extract the kidney stone from the patient130, such as through the percutaneous access point. Although the abovepercutaneous procedure is discussed in the context of using the medicalinstrument 120, in some implementations a percutaneous procedure can beperformed without the assistance of the medical instrument 120. Further,the medical system 100 can be used to perform a variety of otherprocedures.

Minimally invasive surgery offers the potential for video recording ofthe operation, as a camera (e.g., a scope of a medical instrument 120)can be inserted into the body during the surgery. Additional cameras andsensors located outside the body can be used to capture video and/ordata of the patient and medical system 100. For example, operating room(OR) camera(s) can capture video of activities in the operating room,such as movement of an operator's or physician's hands, location of aneedle, replacement of fluidic bags, bleeding of the patient, or thelike. Details such as the number of contrast injections duringfluoroscopy may also be captured by the OR cameras and used to estimatethe amount of radiation exposure to the patient. Audio recorded in thevideos can also be used to help identify phases. For example, somerobotic systems beep or otherwise make an audible noise when lasing isoccurring. The videos may be archived and used later for reasons such ascognitive training, skills assessment, and workflow analysis.

Computer vision, a form of artificial intelligence (AI), allows forquantitative analysis of video by computers for identification ofobjects and patterns. For example, in endoscopic surgery, an AI videosystem can be used for gesture/task classification, skills assessment,tool type recognition, shot/event detection and retrieval. The AI systemcan watch a video of a surgical procedure to track the movement andtiming of instruments used during the procedure. The AI system can usemetrics to track the timing of tools, such as which instrument was usedwhen, and for how long. In addition, the AI system can track the pathwayof the instrument, which can be useful for evaluating procedures oridentifying phases in the procedure. The AI system can determine how farthe tools ranged within the surgical field, which may be correlated tothe quality of surgery as better surgeons tend to handle instruments ina focused area. The AI system can also determine metrics for gaugingmultiple aspects of the medical professionals' performance, includingtheir economy of motion, how often they switched back and forth betweeninstruments, and their efficiency at each step of the procedure.

In the example of FIG. 1 , the medical instrument 120 is implemented asa basket retrieval device. Thus, for ease of discussion, the medicalinstrument 120 is also referred to as “the basket retrieval device 120.”However, the medical instrument 120 can be implemented as various typesof medical instruments including, for example, a scope (sometimesreferred to as an “endoscope”), a needle, a catheter, a guidewire, alithotripter, forceps, a vacuum, a scalpel, a combination of the above,or the like. In some embodiments, a medical instrument is a steerabledevice, while other embodiments a medical instrument is a non-steerabledevice. In some embodiments, a surgical tool refers to a device that isconfigured to puncture or to be inserted through the human anatomy, suchas a needle, a scalpel, a guidewire, and so on. However, a surgical toolcan refer to other types of medical instruments. In some embodiments,multiple medical instruments may be used. For example, an endoscope canbe used with a basket retrieval device 120. In some embodiments, themedical instrument 120 may be a compound device incorporating severalinstruments, such as a vacuum, a basket retrieval device, a scope, orvarious combinations of instruments.

In some embodiments, the medical instrument 120 can include aradio-frequency identification (RFID) chip for identifying the medicalinstrument 120. The medical system 100 can include an RFID reader toread the RFID chip in the medical instrument to aid in identifying theinstrument. Such information can be used to facilitate identifyingprocedures and phases. For example, if the RFID data identifies aninstrument as a needle, the phase may be related to needle insertion,though determining the exact phase may require combining the RFID datawith additional data, such as video, device status, telemetry (e.g.,magnetic tracking, robot data, fluidics data, and/or the like).

The robotic system 110 can be configured to facilitate a medicalprocedure. The robotic system 110 can be arranged in a variety of waysdepending on the particular procedure. The robotic system 110 caninclude one or more robotic arms 112 (robotic arms 112(a), 112(b),112(c)) to engage with and/or control the medical instrument 120 toperform a procedure. As shown, each robotic arm 112 can include multiplearm segments coupled to joints, which can provide multiple degrees ofmovement. In the example of FIG. 1 , the robotic system 110 ispositioned proximate to the patient's 130 lower torso and the roboticarms 112 are actuated to engage with and position the medical instrument120 for access into an access point, such as the urethra of the patient130. With the robotic system 110 properly positioned, the medicalinstrument 120 can be inserted into the patient 130 robotically usingthe robotic arms 112, manually by the physician 160, or a combinationthereof.

The robotic system 110 can also include a base 114 coupled to the one ormore robotic arms 112. The base 114 can include a variety of subsystems,such as control electronics, a power source, pneumatics, an opticalsource, an actuator (e.g., motors to move the robotic arm), controlcircuitry, memory, and/or a communication interface. In someembodiments, the base 114 includes an input/output (I/O) device 116configured to receive input, such as user input to control the roboticsystem 110, and provide output, such as patient status, medicalinstrument location, or the like. The I/O device 116 can include acontroller, a mouse, a keyboard, a microphone, a touchpad, other inputdevices, or combinations of the above. The I/O device can include anoutput component, such as a speaker, a display, a haptic feedbackdevice, other output devices, or combinations of the above. In someembodiments, the robotic system 110 is movable (e.g., the base 114includes wheels) so that the robotic system 110 can be positioned in alocation that is appropriate or desired for a procedure. In otherembodiments, the robotic system 110 is a stationary system. Further, insome embodiments, the robotic system 110 is integrated into the table150.

The robotic system 110 can be coupled to any component of the medicalsystem 100, such as the control system 140, the table 150, the imagingsensor 180, and/or the medical instruments 120. In some embodiments, therobotic system is communicatively coupled to the control system 140. Inone example, the robotic system 110 can receive a control signal fromthe control system 140 to perform an operation, such as to position arobotic arm 112 in a particular manner, manipulate a scope, and so on.In response, the robotic system 110 can control a component of therobotic system 110 to perform the operation. In another example, therobotic system 110 can receive an image from the scope depictinginternal anatomy of the patient 130 and/or send the image to the controlsystem 140 (which can then be displayed on the control system 140).Further, in some embodiments, the robotic system 110 is coupled to acomponent of the medical system 100, such as the control system 140, toreceive data signals, power, and so on. Other devices, such as othermedical instruments, intravenous bags, blood packs or the like can alsobe coupled to the robotic system 110 or other components of the medicalsystem 100 depending on the medical procedure being performed.

The control system 140 can be configured to provide variousfunctionality to assist in performing a medical procedure. In someembodiments, the control system 140 can be coupled to the robotic system110 and operate in cooperation with the robotic system 110 to perform amedical procedure on the patient 130. For example, the control system140 can communicate with the robotic system 110 via a wireless or wiredconnection (e.g., to control the robotic system 110, the basketretrieval device 120, receive an image(s) captured by a scope, etc.),control the flow of fluids through the robotic system 110 via one ormore fluid channels, provide power to the robotic system 110 via one ormore electrical connections, provide optical signals to the roboticsystem 110 via one or more optical fibers or other components, and soon. Further, in some embodiments, the control system 140 can communicatewith a scope to receive sensor data. Moreover, in some embodiments, thecontrol system 140 can communicate with the table 150 to position thetable 150 in a particular orientation or otherwise control the table150.

As shown in FIG. 1 , the control system 140 includes various I/O devicesconfigured to assist the physician 160 or others in performing a medicalprocedure. In some embodiments, the control system 140 includes an inputdevice 146 that is employed by the physician 160 or another user tocontrol the basket retrieval device 120. For example, the input device146 can be used to navigate the basket retrieval device 120 within thepatient 130. The physician 160 can provide input via the input device146 and, in response, the control system 140 can send control signals tothe robotic system 110 to manipulate the medical instrument 120.

In some embodiments, the input device 146 is a controller similar to agame controller. The controller can have multiple axes and buttons thatcan be used for controlling the robotic system 110. While the inputdevice 146 is illustrated as a controller in the example of FIG. 1 , theinput device 146 can be implemented as a variety or combination of typesof I/O devices, such as a touchscreen/pad, a mouse, a keyboard, amicrophone, a smart speaker, etc. As also shown in FIG. 1 , the controlsystem 140 can include the display 142 to provide various informationregarding a procedure. For example, the control system 140 can receivereal-time images that are captured by a scope and display the real-timeimages via the display 142. Additionally or alternatively, the controlsystem 140 can receive signals (e.g., analog, digital, electrical,acoustic/sonic, pneumatic, tactile, hydraulic, etc.) a medical monitorand/or a sensor associated with the patient 130, and the display 142 canpresent information regarding the health of the patient 130 and/or anenvironment of the patient 130. Such information can include informationthat is displayed via a medical monitor including, for example, a heartrate (e.g., electrocardiogram (EGG), heart rate variability (HRV),etc.), blood pressure/rate, muscle bio-signals (e.g., electromyography(EMS)), body temperature, oxygen saturation (e.g., SpO₂), carbon dioxide(CO₂), brainwave (e.g., electroencephalogram (EEG)), environmentaltemperature, and so on.

FIG. 1 also shows various anatomy of the patient 130 relevant to certainaspects of the present disclosure. In particular, the patient 130includes kidneys 170 fluidly connected to a bladder 171 via ureters 172,and a urethra 173 fluidly connected to the bladder 171. As shown in theenlarged depiction of the kidney 170, the kidney includes calyxes 174(e.g., major and minor calyxes), renal papillae (including the renalpapilla 176, also referred to as “the papilla 176”), and renal pyramids(including the renal pyramid 178). In these examples, a kidney stone 165is located in proximity to the papilla 176. However, the kidney stonecan be located at other locations within the kidney 170.

As shown in FIG. 1 , to remove the kidney stone 165 in the exampleminimally-invasive procedure, the physician 160 can position the roboticsystem 110 at the foot of the table 150 to initiate delivery of themedical instrument 120 into the patient 130. In particular, the roboticsystem 110 can be positioned within proximity to a lower abdominalregion of the patient 130 and aligned for direct linear access to theurethra 173 of the patient 130. From the foot of the table 150, therobotic arm 112(B) can be controlled to provide access to the urethra173. In this example, the physician 160 inserts a medical instrument 120at least partially into the urethra along this direct linear access path(sometimes referred to as “a virtual rail”). The medical instrument 120can include a lumen configured to receive the scope and/or basketretrieval device, thereby assisting in insertion of those devices intothe anatomy of the patient 130.

Once the robotic system 110 is properly positioned and/or the medicalinstrument 120 is inserted at least partially into the urethra 173, thescope can be inserted into the patient 130 robotically, manually, or acombination thereof. For example, the physician 160 can connect themedical instrument 120 to the robotic arm 112(C). The physician 160 canthen interact with the control system 140, such as the input device 146,to navigate the medical instrument 120 within the patient 130. Forexample, the physician 160 can provide input via the input device 146 tocontrol the robotic arm 112(C) to navigate the basket retrieval device120 through the urethra 173, the bladder 171, the ureter 172, and up tothe kidney 170.

The control system 140 can include various components (sometimesreferred to as “subsystems”) to facilitate its functionality. Forexample, the control system 140 can include a variety of subsystems,such as control electronics, a power source, pneumatics, an opticalsource, an actuator, control circuitry, memory, and/or a communicationinterface. In some embodiments, the control system 140 includes acomputer-based control system that stores executable instructions, thatwhen executed, implement various operations. In some embodiments, thecontrol system 140 is movable, such as that shown in FIG. 1 , while inother embodiments, the control system 140 is a stationary system.Although various functionality and components are discussed as beingimplemented by the control system 140, any of this functionality and/orcomponents can be integrated into and/or performed by other systemsand/or devices, such as the robotic system 110 and/or the table 150.

The medical system 100 can provide a variety of benefits, such asproviding guidance to assist a physician in performing a procedure(e.g., instrument tracking, patient status, etc.), enabling a physicianto perform a procedure from an ergonomic position without the need forawkward arm motions and/or positions, enabling a single physician toperform a procedure with one or more medical instruments, avoidingradiation exposure (e.g., associated with fluoroscopy techniques),enabling a procedure to be performed in a single-operative setting,providing continuous suction to remove an object more efficiently (e.g.,to remove a kidney stone), and so on. Further, the medical system 100can provide non-radiation-based navigational and/or localizationtechniques to reduce physician exposure to radiation and/or reduce theamount of equipment in an operating room. Moreover, the medical system100 can divide functionality into the control system 140 and the roboticsystem 110, each of which can be independently movable. Such a divisionof functionality and/or movability can enable the control system 140and/or the robotic system 110 to be placed at locations that are optimalfor a particular medical procedure, which can maximize working areaaround the patient, and/or provide an optimized location for a physicianto perform a procedure. For example, many aspects of the procedure canbe performed by the robotic system 110 (which is positioned relativelyclose to the patient) while the physician manages the procedure from thecomfort of the control system 140 (which can be positioned farther way).

In some embodiments, the control system 140 can function even if locatedin a different geographic location from the robotic system 110. Forexample, in a tele-health implementation, the control system 140 isconfigured to communicate over a wide area network with the roboticsystem 110. In one scenario, a physician 160 may be located in onehospital with the control system 140 while the robotic system 110 islocated in a different hospital. The physician may then perform themedical procedure remotely. This can be beneficial where remotehospitals, such as those in rural areas, have limited expertise inparticular procedures. Those hospitals can then rely on more experiencedphysicians in other locations. In some embodiments, a control system 140is able to pair with a variety of robotic systems 110, for example, byselecting a specific robotic system and forming a secure networkconnection (e.g., using passwords, encryption, authentication tokens,etc.). Thus, a physician in one location may be able to perform medicalprocedures in a variety of different locations by setting up aconnection with robotic systems 110 located at each of those differentlocations.

In some embodiments, the robotic system 110, the table 150, the medicalinstrument 120, the needle and/or the imaging sensor 180 arecommunicatively coupled to each other over a network, which can includea wireless and/or wired network. Example networks include one or morepersonal area networks (PANs), one or more local area networks (LANs),one or more wide area networks (WANs), one or more Internet areanetworks (IANs), one or more cellular networks, the Internet, etc.Further, in some embodiments, the control system 140, the robotic system110, the table 150, the medical instrument 120, and/or the imagingsensor 180 are connected for communication, fluid/gas exchange, powerexchange, and so on via one or more support cables.

Although not illustrated in FIG. 1 , in some embodiments the medicalsystem 100 includes and/or is associated with a medical monitorconfigured to monitor the health of the patient 130 and/or anenvironment in which the patient 130 is located. For example, a medicalmonitor can be located in the same environment where the medical system100 is located, such as within an operating room. The medical monitorcan be physically and/or electrically coupled to one or more sensorsthat are configured to detect or determine one or more physical,physiological, chemical, and/or biological signals, parameters,properties, states and/or conditions associated with the patient 130and/or the environment. For example, the one or more sensors can beconfigured to determine/detect any type of physical properties,including temperature, pressure, vibration, haptic/tactile features,sound, optical levels or characteristics, load or weight, flow rate(e.g., of target gases and/or liquid), amplitude, phase, and/ororientation of magnetic and electronic fields, constituentconcentrations relating to substances in gaseous, liquid, or solid form,and/or the like. The one or more sensors can provide the sensor data tothe medical monitor and the medical monitor can present informationregarding the health of the patient 130 and/or the environment of thepatient 130. Such information can include information that is displayedvia a medical monitor including, for example, a heart rate (e.g., ECG,HRV, etc.), blood pressure/rate, muscle bio-signals (e.g., EMG), bodytemperature, oxygen saturation (e.g., SpO₂), CO₂, brainwave (e.g., EEG),environmental temperature, and so on. In some embodiments, the medicalmonitor and/or the one or more sensors are coupled to the control system140 and the control system 140 is configured to provide informationregarding the health of the patient 130 and/or the environment of thepatient 130.

Urinary Stone Capture

FIGS. 2A-2B illustrate a perspective view of the medical system 100while performing a urinary stone capture procedure. In these examples,the medical system 100 is arranged in an operating room to remove akidney stone from the patient 130. In many instances of such aprocedure, the patient 130 is positioned in a modified supine positionwith the patient 130 slightly tilted to the side to access the posterioror side of the patient 130. The urinary stone capture procedure may alsobe performed with the patient in a regular supine position, as show inin FIG. 1 . Although FIGS. 2A-2B illustrate the use of the medicalsystem 100 to perform a minimally-invasive procedure to remove a kidneystone from the patient 130, the medical system 100 can be used to removea kidney stone in other manners and/or to perform other procedures.Further, the patient 130 can be arranged in other positions as desiredfor a procedure. Various acts are described in FIGS. 2A-2B andthroughout this disclosure as being performed by the physician 160. Itshould be understood that these acts can be performed directly by thephysician 160, indirectly by the physician with the aid of the medicalsystem 100, by a user under the direction of the physician, by anotheruser (e.g., a technician), and/or any other user.

Although particular robotic arms of the robotic system 110 areillustrated as performing particular functions in the context of FIGS.2A-2B, any of the robotic arms 112 can be used to perform the functions.Further, any additional robotic arms and/or systems can be used toperform the procedure. Moreover, the robotic system 110 can be used toperform other parts of the procedure.

As shown in FIG. 2A, the basket retrieval device 120 is maneuvered intothe kidney 170 to approach the urinary stone 165. In some scenarios, thephysician 160 or other user uses the input device 146 to directlycontrol movement of the basket retrieval device 120. Such directlycontrolled movement can include insertion/retraction, flexing the basketretrieval device 120 left or right, rotation, and/or regular open/closeof the basket. Using various movements, the basket retrieval device 120is placed close to the stone.

In some embodiments, a laser, shock wave device, or other device is usedto break up the stone. The laser or other device may be incorporatedinto the basket retrieval device 120 or may be a separate medicalinstrument. In some situations, the stone 165 is small enough thatbreaking up the stone into smaller pieces is not needed.

As shown in FIG. 2B, the open basket is maneuvered to surround theurinary stone 165 or a smaller piece of the urinary stone. The basketretrieval device 120 is then withdrawn from the kidney 170 and then outof the patient's body.

If additional stones (or large pieces of a broken-up stone 165) exist,the basket retrieval device 120 may be reinserted into the patient tocapture the remaining large pieces. In some embodiments, a vacuuminstrument can be used to facilitate removal of the pieces. In somesituations, the stone pieces may be sufficiently small that they can bepassed by the patient naturally.

Phase Segmentation and Phase Recognition

Automated surgical workflow analysis can be used to detect differentphases in a procedure and to assess surgical skill and proceduralefficiency. Data collected during procedures (e.g. video data) can besegmented into multiple sections using, for example, machine learningmethods, including but not limited to, a hidden Markov model (HMM) and along-term-short-memory (LTSM) network.

In surgical phase segmentation, captured medical procedure data isautomatically segmented into phases, using input data from the operatingroom to identify the phases. Segmentation may be done in real-timeduring the procedure or performed post-operatively on recorded data. Inone embodiment, the surgical data can be preprocessed using dynamic timewarping to divide the phases into equal comparable segments. The inputdata can consist of instrument signals, annotations, tracking ofinstruments (e.g. EM), or information obtained from videos.

Recognition of surgical workflow can be done at different granularitylevels, depending on the procedure. It can be done on phases and steps(higher level), or gestures and activities (lower level). Surgical phaserecognition can be performed on time series, kinematic data, and videodata using machine learning approaches such as HMMs, Gaussian MixtureModels (GMMs), and Support Vector Machines (SVMs), as well as deeplearning-based approaches for phase recognition from video data usingConvolutional Neural Networks (CNNs). For surgical gesture and activityrecognition, similar methods (SVMs, Markov models) can be used on mainlyvideo data or a combination of video and kinematic data, as well as morerecent deep-learning based methods such as CNNs that can be used forrecognition of tool presence, tasks, and activities in video data. Phasesegmentation can use multiple data sources to segment the case data todifferent subtasks as shown in FIG. 3 or use a single data source, suchas video, to classify the current phase as shown in FIG. 4A. In FIG. 4B,additional data (e.g. sensor data or UI data) can then be incorporatedto further refine the output produced by the control system 140.

In FIG. 3 , the control system 140 receives various input data from themedical system 100. Such input can include video data 305 captured bythe imaging sensor 180, robotic sensor data 310 from one or more sensorsof the robotic system 110, and user interface (UI) data received fromthe input device 146.

Video data 305 can include video captured from scopes deployed within apatient, video captured from cameras in the operating room, and/or videocaptured by cameras of the robotic system 110. Robotic sensor data 310can include kinematic data from the robotic system 110 (e.g., usingvibration, accelerometer, positioning, and/or gyroscopic sensors),device status, temperature, pressure, vibration, haptic/tactilefeatures, sound, optical levels or characteristics, load or weight, flowrate (e.g., of target gases and/or liquid), amplitude, phase, and/ororientation of magnetic and electronic fields, constituentconcentrations relating to substances in gaseous, liquid, or solid form,and/or the like. UI data 315 can include button presses, menuselections, page selections, gestures, voice commands, and/or the likemade by the user and captured by input devices of the medical system100. Patient sensor data, such as those described in FIG. 1 above, mayalso be used as an input to the control system 140.

The control system 140 can analyze the video data 305 (e.g., usingmachine learning algorithms), as well as using robotic sensor data 310and UI data 315 to identify phases of the a medical procedure. In oneexample, a medical procedure such as ureteroscopy includes several tasks(e.g., Task 1-Task 5). Each task may be performed in one or more phasesof the medical procedure. In the example shown in FIG. 3 , Task 1 isperformed in phase 1. Task 2 is performed in phase 2 and 4. Task 3 isperformed in phase 3 and phase 5. Task 4 is performed din phase 6 and 8.Task 5 is performed in phase 7. Time 1 (T1) denotes the time taken tocomplete phase 1, Time 2 (T2) denotes the time taken to complete phase2, and time 3 (T3) denotes the time taken to complete phase 3. Otherprocedures may have a different number of tasks and/or a differentnumber of phases.

For robotic procedures where there are manual and automated task,surgical phase detection can be used to make the transition betweenmanual and automated tasks automatic and seamless. For example, T1 maycorrespond to a manual task, T2 may be an automated task, and T3 mayagain be a manual task. In one embodiment, when the target selectionphase is active, the target selection step can be autonomously performedby the robot driving the scope. Alternatively, the user can perform siteselection by picking a point on the skin using an EM marker, and therobot can autonomously align the needle to the target insertiontrajectory.

FIG. 4A illustrates a block diagram of the control system 140 configuredto utilize machine learning to generate outputs from video data from amedical procedure, according to certain embodiments. In someembodiments, the control system 140 is configured to process the videodata 305 first, using machine learning algorithms. In one embodiment,video data 305 is processed by a CNN 405 to generate output 412 toidentify features recorded in the video, such as surgical tools,stone(s), human anatomy (e.g., papilla), or the like. Such identifiedfeatures 415 may be provided as input to a recurrent neural networks(RNN) 410, along with the original video. The RNN 410 can then processthe video data 305 and the identified features 415 to generate output412 to identify phases 420 in a medical procedure.

Supplemental data such as robotic sensor data 310 or UI data 315 maythen be used to further refine (e.g., increase accuracy or increase thenumber of identifications) the identified features 415 and theidentified phases 420. In other embodiments, the robotic sensor data 310and/or UI data 315 can be used prior to the processing of the video data305 by the control system 140 to narrow the possible options consideredby the control system 140. For example, the supplemental data can beused to identify a specific procedure, which narrows the universe ofpossible tasks and phases to those corresponding to the specificprocedure. The control system 140 may then limit the identified features415 and identified phases 420 to those that correspond to the specificprocedure. For example, if a task is initially identified in the videodata 305 by the control system 140, but that task is not associated withthe specific procedure, the control system 140 may reprocess the videountil the task is re-identified as a task corresponding to the specificprocedure.

After completing processing of the video data 305, the control system140 may generate an annotated video that includes the identifiedfeatures 415 and/or identified phases 420. Such annotations may bestored as part of the video (e.g., in the same video file), meta-datastored alongside the video, in a database, and/or other data format.

By creating meta-data enhanced video, the video becomes easier to usefor reviewing medical procedures. For example, a viewer can jump forwardor backward to specific phase of interest rather than manually searchingfor when a specific phase occurred. In addition, multiple videos can bemore easily processed to aggregate data and generate metrics. Forexample, multiple videos can be searched for instances of a particularphase (e.g., needle insertion or stone capture), and analyzed togenerate metrics about the that phase (e.g., success rates, averageattempts, number of attempts, etc.).

While FIG. 4A shows video data 305 being processed by the control system140, other types of data can be processed by the control system 140,serially or in tandem with each other. For example, such data caninclude instrument positioning as measure by electromagnetic trackingsensors, robotic system 110 data such as how far the scope is inserted,how the scope is articulated, if basket is open or closed, how far thebasket is inserted, and/or the connection status of the robotic system.The data can be provided as input to a single neural network or tomultiple neural networks. For example, each different type of sensor(e.g., video, device status, telemetry such as: magnetic tracking; robotdata; and/or fluidics data) may have its own network and the outputs ofthe networks may be concatenated before the final phase classificationlayer to obtain a single phase prediction.

FIG. 4B illustrates one such embodiment where different types of datafrom different devices and/or sensors are processed by different neuralnetworks. Video data 305 can be processed by a first neural network 425(e.g., CNN and/or RNN as described in FIG. 4A), robotic sensor data 310can be processed by a second neural network 430, and UI data can beprocessed by a third neural network 435. The outputs from the differentneural networks may then be combined to generate an output 412 (e.g.,phase prediction) for the medical system 100.

Phase Identification Process

FIG. 5 is a flow diagram of a phase identification process 500,according to certain embodiments. The phase identification process 500can be performed by the control system 140 or by another component ofthe medical system 100 of FIG. 1 . While the following describes onepossible sequence to the process, other embodiments can perform theprocess in a different order or may include additional steps or mayexclude one or more of the steps described below.

At block 505, the control system 140 identifies an input from a UI ofthe robotic system. For example, the input may be received from an inputdevice 146 such as controller or touch screen. Possible inputs caninclude selection of a procedure phase or of a UI screen associated witha specific procedure phase. For example, a first screen may list optionsfor a first procedure, while a second screen may list options for secondprocedure. If the user is making selections on the first screen, thenthose selections indicate the user will be performing the firstprocedure. If the user is making selections on the second screen, thenthose selections indicate the user will be performing the secondprocedure. Thus, by organizing the screens of the UI to correspond toparticular phases, the control system 140 can obtain phase informationbased on the user's selections. In another example, one embodiment ofthe medical system 100 can include a UI with a first screen showingselectable stone management procedures, such as ureteroscopy,percutaneous access or mini percutaneous nephrolithotomy (PCNL). If theuser selects ureteroscopy, the control system 140 can determine that thephases are related to ureteroscopy (e.g., basketing, lasing and/orsurveying the kidney). Likewise, selecting the other stone managementprocedures indicates the phases are related to the correspondingprocedures.

At block 510, the control system 140 determines a procedure from a setof procedures based on at least one of the UI input and sensor data. Asdescribed above, the input from the UI interface can be used to identifythe current possible procedure phase. In addition, robotic sensor datacan also be used to identify the procedure. For example, if an arm ofthe robotic system 110 is determined to be approaching the patient whileholding a surgical instrument, the control system 140 may determine thatthe current procedure is related to insertion of a medical instrument.

At block 515, the control system 140 can narrow the set of identifiableprocedure phases to a subset of the procedure phases based on thedetermined procedure. For example, lasing may be associated with tasksor phases such as activating a laser or stopping the laser. Basketingmay be associated with tasks or phases such as capturing a stone orretracting the basket. Insertion of a medical instrument 120 may beassociated with aligning the instrument with the target and insertingthe instrument into the target site. In one example, if the controlsystem 140 determines the current procedure is basketing duringureteroscopy, the control system 140 can narrow the possible phases tocapturing a stone or retracting the basket.

At block 520, the control system 140 can determine a position of arobotic manipulator (e.g., robotic arm 112) from sensor data of therobotic system 110. As described in FIG. 3 , various types of sensorscan be used to generate sensor data, which may then be used to determinethe position.

At block 525, the control system 140 can perform an analysis of acaptured video. In some embodiments, such as those described in FIG. 4A,machine learning algorithms are used to perform the analysis andgenerate output such as identified features and provisionalidentification of phases. Outputs can include identification of physicalobjects such as surgical tools or parts of the anatomy. For example, ifthe control system 140 identifies a ureter in the captured video, thatindicates the phase is not related to percutaneous access. Similarly,identifying papilla indicates the phase is not related to basketing.Identification of other types of anatomy can similarly be used toeliminate the possibility of certain phases.

At block 530, the control system 140 can identify a phase from thesubset of procedure phases based on at least the position of the roboticmanipulator and the performed analysis. For example, if the controlsystem 140 is receiving basketing inputs via a controller, the controlsystem 140 can determine that the phase is one of the basketing phases.In addition, if the performed analysis identifies that the capturedvideo is showing a basket approaching a broken-up kidney stone, thecontrol system 140 can determine that the current phase is capturing thestone. In another example, if the performed analysis identifies that thecaptured video is showing a basket withdrawing from the broken-up kidneystone, the control system 140 can determine that the current phase isretracting the basket into a sheath. In a further example, kinematicdata from the robotic system 110 may indicate a medical instrument isbeing withdrawn from within the patient and the control system 140 maydetermine that the current phase is retracting the basket into a sheath.

At block 535, the control system 140 can generate a video marker for theidentified phase for the captured video. The video marker may beembedded as meta-data in the same file as the video, as a separate fileassociated with the video file, as meta-data stored in a database forvideo annotations, or the like.

In some embodiments, the video file is annotated such that viewers ofthe video file can jump to specific phases in the video. For example,the video may be divided into chapters or segments corresponding to thedifferent phases. In one embodiment, a seek bar for the video may bemarked with colored segments corresponding to different phases, whereeach phase is denoted by a different color.

At block 550, the control system 140 can determine if the end of thevideo is reached. If yes, the process 500 can end. If no, the process500 can loop back to block 520 to continue identifying additionalphases. For example, the process 500 may loops once, twice, three times,or more to identify a first phase, a second phase, a third phase, ormore phases. Subsequently, the captured video may end up with one ormore video markers, depending on the number of phases identified.

Triggering Automated Actions

FIG. 6 is a flow diagram of a triggering process 600 for automatedrobotic action, according to certain embodiments. The triggering process600 can be performed by the control system 140 or by another componentof the medical system 100 of FIG. 1 . While the following describes onepossible sequence to the process, other embodiments can perform theprocess in a different order or may include additional steps or mayexclude one or more of the steps described below.

At block 605, the control system 140 can determine a status of a roboticmanipulator (e.g., robotic arm 112) from sensor data (e.g., kinematicdata) of the robotic system 110. As described in FIG. 3 , various typesof sensors can be used to generate sensor data, which may then be usedto determine the position or other status of the robotic manipulator.

At block 610, the control system 140 can determine an input forinitiating an action of the robotic manipulator. For example, the inputmay be from a user manipulating a controller to control a basketingdevice. In another example, the input may be a screen selection or amenu selection on a UI of the medical system 100.

At block 615, the control system 140 can perform an analysis of acaptured video. In some embodiments, such as those described in FIG. 4A,machine learning algorithms are used to perform the analysis andgenerate output such as identified features and provisionalidentification of phases.

At block 620, the control system 140 can identify a phase of medicalprocedure based at least on the status of the manipulator, theidentified input, and the performed analysis. For example, if thecontrol system 140 is receiving basketing inputs via a controller, thecontrol system 140 can determine that the phase is one of the basketingphases. In addition, if the performed analysis identifies that thecaptured video is showing a basket approaching a broken-up kidney stone,the control system 140 can determine that the current phase is capturingthe stone. In another example, if the performed analysis identifies thatthe captured video is showing a basket withdrawing from the broken-upkidney stone, the control system 140 can determine that the currentphase is retracting the basket into a sheath. In a further example,kinematic data from the robotic system 110 may indicate a medicalinstrument is being withdrawn from within the patient and the controlsystem 140 may determine that the current phase is retracting the basketinto a sheath.

At block 625, the control system 140 can trigger an automatic action ofthe robotic system 110 based on the identified phase. The triggeredaction can vary based on the type of procedure being performed. Somepossible actions are shown in blocks 630, 635, and 640. At block 630,the robotic system 110 performs an action during ureteroscopy lasing. Atblock 635, the robotic system 110 performs an action during insertion ofa medical instrument, such as a needle. At block 635, the robotic system110 performs an action during ureteroscopy basketing. After causing anaction by the robotic system 110, the triggering process 600 can end.FIG. 7 describes additional detail on specific actions that may betriggered.

FIG. 7 is a diagram showing different types of triggered actions of therobotic system 110, according to certain embodiments. The actions may betriggered in response to identifying a current phase of an operation oridentifying a user action. In some embodiments, the actions may be fullyautomatic and performed without needing additional input from a user. Inother embodiments, the actions may be partially automated, requiringconfirmation from the user before being performed by the robotic system110. Different combinations of the phases may be performed based on theprocedure being performed by the robotic system 110. Some exampleprocedures include (retrograde) ureteroscopy, percutaneousnephrostolithotomy (PCNL), mini-PCNL or the like. For example,ureteroscopy can include a surveying phase (not shown), a lasing phase,and a basking phase. PCNL can include a percutaneous access phase, asurveying phase, a lasing phase and a basketing phase. Mini-PCNL caninclude additional alignment and/or aspiration phases.

For example, during lasing 705, actions that can be triggered includeapplying a laser to a stone 710 and stopping the laser when not pointedat the stone 715. In one scenario, the robotic system 110, using varioussensors (e.g., a camera), can detect when the laser is pointed at thestone. It may then determine the size of the stone, for example, byusing machine learning algorithms that have been trained usingrecordings of previous ureteroscopy procedures or by using traditionalcomputer vision algorithms (e.g., comparing the known size of the basketwith the size of the stone). Based on the determined size, the roboticsystem 110 can then determine an initial lasing time based on recordedlasing times for similar sized and/or types of stones. The roboticsystem 110 can then stop the laser after the determined lasing time orif it detects that the stone has broken up. In other scenarios, the usermay provide additional input, such as setting the lasing time orproviding permission for the laser to be activated by the roboticsystem.

In another scenario, applying the laser may be triggered by the userwhile the stopping the laser is triggered automatically by the roboticsystem 110. For example, the robotic system 110, using its sensors, candetect when the targeting of the laser drifts from the stone or isotherwise not centered on the stone and stop the laser in response.

In another example, during basketing 725, actions that can be triggeredinclude capturing a stone inside a basket 730 and retracing the basketinto a sheath 735. In one scenario, the robotic system 110 can triggeractuation of the basket 730 when it detects that the basket 730 isaligned with the stone and within a specified distance. The basket 730can then be actuated to capture the stone. The robotic system 110, usingits sensors (e.g., camera or pressure sensors) can then determine if thestone is captured inside the basket 730 and trigger the retraction ofthe basket into the sheath 735. The user may then retract the sheathfrom the patient's body, thereby removing the stone. In another example,during percutaneous access 740, actions that can be triggered includetarget (calyx) selection 745, insertion site selection 750, and needleinsertion 755 into the target site. In one scenario, the robotic system110 can determine the target and the insertion site at the target (e.g.,marked by the user or identified by the system). The robotic system 110may then wait for confirmation from the user to proceed. After receivingconfirmation, the robotic system 110 may then insert the needle (orother instrument) into the target site.

In another example, during a mini_PCNL procedure, additional phases caninclude robotic alignment with a PCNL sheath 765 and lasing a stone withactive irrigation and aspiration 770. Triggered actions in these phasescan include aligning an instrument with the PCNL sheath and increasingaspiration. For example, if the robotic system 110 detects an increasein stone fragments during lasing or otherwise greater dusting thatlimits visibility, the robotic system 110 can increase aspiration orsuction to remove more of the stone fragments. Once visibility or fieldof view increases, the robotic system 110 can reduce aspiration.

While the above has discussed some examples and scenarios of automaticactions of the robotic system 110 that can be triggered based on theidentified phase, the triggerable actions are not limited to the actionsdiscussed above. The robotic system 110 may be programmed to performother triggerable actions based on the needs of the users and patients.

Evaluating Tasks Performed During Phases

FIG. 8 is a flow diagram of an evaluation process 800 for tasksperformed during identified phases, according to certain embodiments.The evaluation process 800 can be performed by the control system 140 orby another component of the medical system 100 of FIG. 1 . While thefollowing describes one possible sequence to the process, otherembodiments can perform the process in a different order or may includeadditional steps or may exclude one or more of the steps describedbelow.

At block 805, the control system 140 can determine a status of a roboticmanipulator (e.g., robotic arm 112) from sensor data of the roboticsystem 110. As described in FIG. 3 , various types of sensors can beused to generate sensor data, which may then be used to determine theposition or other status of the robotic manipulator.

At block 810, the control system 140 can determine an input forinitiating an action of the robotic manipulator. For example, the inputmay be from a user manipulating a controller to control a basketingdevice. In another example, the input may be a screen selection or amenu selection on a UI of the medical system 100.

At block 815, the control system 140 can perform an analysis of acaptured video. In some embodiments, such as those described in FIG. 4A,machine learning algorithms are used to perform the analysis andgenerate output such as identified features and provisionalidentification of phases.

At block 820, the control system 140 can identify a phase of medicalprocedure based at least on the status of the manipulator, theidentified input, and the performed analysis. For example, if thecontrol system 140 is receiving basketing inputs via a controller, thecontrol system 140 can determine that the phase is one of the basketingphases. In addition, if the performed analysis identifies that thecaptured video is showing a basket approaching a broken-up kidney stone,the control system 140 can determine that the current phase is capturingthe stone. In another example, if the performed analysis identifies thatthe captured video is showing a basket withdrawing from the broken-upkidney stone, the control system 140 can determine that the currentphase is retracting the basket into a sheath. In a further example,kinematic data from the robotic system 110 may indicate a medicalinstrument is being withdrawn from within the patient and the controlsystem 140 may determine that the current phase is retracting the basketinto a sheath.

At block 825, the control system 140 can generate an evaluation of theidentified phase based on one or more metrics. The evaluated phase canvary based on the type of procedure being performed. Some possiblephases are shown in blocks 830, 835, and 840. At block 830, the controlsystem 140 evaluates a ureteroscopy lasing phase. At block 835, thecontrol system 140 evaluates a medical instrument insertion phase. Atblock 840, the control system 140 evaluates a ureteroscopy basketingphase. Some specific examples of various evaluations are describedbelow.

FIG. 9 is a flow diagram of a scoring process 900 for medical tasks,according to certain embodiments. The scoring process 900 can beperformed by the control system 140 or by another component of themedical system 100 of FIG. 1 . While the following describes onepossible sequence to the process, other embodiments can perform theprocess in a different order or may include additional steps or mayexclude one or more of the steps described below.

At block 905, the control system 140 counts a number of times a firstprocedure task is performed. At block 910, the control system 140 countsa number of times a second procedure task is performed. At block 915,the control system 140 determines a ratio of the counted number for thefirst procedure task to the counted number for the second proceduretask. At block 920, the control system 140 can compare the determinedratio with a historical ratio. For example, the historical ratio may begenerated by analyzing historical records for the same procedure todetermine a mean or median ratio.

In one example, during ureteroscopy basketing, the control system 140can count a number of basket operations and count a number ofureteroscope retractions. The control system 140 can then determine aratio of the number of basket operations to the number of ureteroscoperetractions and compare the determined ratio with other ratios fromprevious ureteroscopy basketing procedures.

In one example for ureteroscopy driving, the control system 140 cancount a number of times a user drives a scope manually and a number oftimes the user drives the scope robotically. Manual driving is generallyused for surveying the kidney. Meanwhile, a scope is typically docked tothe robotic system in order to perform basketing. The control system 140can then determine a ratio of the number of times the user drives ascope manually to the number of times the user drives the scoperobotically and compare the determined ratio with other recorded ratiosfrom previous ureteroscopy procedures. This ratio can measure the levelof adaptation of the user to robotic ureteroscopy.

In another example, during ureteroscopy lasing, the control system 140can count a lasing time for a stone and determine a size and/or type ofthe stone. The control system 140 can then determine a ratio of thelasing time for the stone with the size of the stone and compare thedetermined ratio with previous ratios from other operations. Bydetermining the type of the stone (e.g., uric acid, calcium oxalatemonohydrate, struvite, cystine, brushite, etc.), the control system 140can aggregate statics across surgical operations based on the type ofthe stone. For example, lasing duration and procedure duration can bebroken out by type of stone.

At block 925, the control system 140 can generate an output of thecomparison. Such an output can be a report, visual indicator, guide,score, graph, or the like. For example, the control system 140 mayindicate that the user is performing at, below, or above a median oraverage value for the ratio in comparison to recorded ratios fromprevious operations. In some embodiments, the output may compare thecurrent user with records of previous operations by that user to trackthe user's personal performance. In some embodiments, the output maycompare the user with other medical professionals.

In one embodiment, the output can include a real-time indicator showinghow the user's current performance compares to previous operations. Suchan output can aid the user during surgery by, for example, giving theuser input on how long to perform lasing based on the size of the stone.Other outputs can provide other relevant information for the user.

Various types of procedure tasks can be evaluated using the scoringprocess 900. For example, some ratios can include number of basketoperations to number of ureteroscope retractions, number of times a userdrives a scope manually to number of times the user drives the scoperobotically, and lasing time for a stone to size of the stone.

FIG. 10 is a flow diagram of another scoring process for medical tasks,according to certain embodiments. The scoring process 1000 can beperformed by the control system 140 or by another component of themedical system 100 of FIG. 1 . While the following describes onepossible sequence to the process, other embodiments can perform theprocess in a different order or may include additional steps or mayexclude one or more of the steps described below.

At block 1005, the control system 140 can count a first procedure task.At block 1010, the control system 140 can compare the first proceduretask with a historical count for the first procedure. For example,during ureteroscopy driving, the control system 140 may count the numberof times a user attempts to insert a needle until the user succeeds andcompares that count with the recorded needle insertion attempts fromprevious percutaneous needle insertion operations.

In another example, during percutaneous needle insertion, the controlsystem 140 may count the time taken to survey a kidney before selectinga target calyx for percutaneous access and compare the counted time withrecorded times from previous percutaneous needle insertion operations.The control system 140, during percutaneous needle insertion, may alsocount the number of times an automated alignment of the roboticmanipulator with a catheter is initiated and compare the counted numberof times with the recorded automated alignment numbers from previousoperations.

During mini-PCNL alignment, the control system 140 may count a number oftimes an automated alignment of an end effector of the roboticmanipulator with a catheter or sheath is initiated and compare thecounted number of times with recorded automated alignment numbers fromprevious operations. In another example, the control system 140 duringureteroscopy lasing may count a number of times a view of the videocapture device becomes occluded by dust from fragmentation of a stoneand compare the counted number of times with recorded number of dustocclusions from previous operations.

At block 1015, the control system 140 can generate an output of thecomparison. Such an output can be a report, visual indicator, guide,score, graph, or the like. For example, the control system 140 mayindicate that the user is performing at, below, or above a median oraverage in comparison to recorded metrics from previous operations. Insome embodiments, the output may compare the current user with recordsof previous operations by that user to track the user's personalperformance. In some embodiments, the output may compare the user withother users.

In one embodiment, the output can include a real-time indicator showinghow the user's current performance compares to previous operations. Suchan output can aid the user during surgery by, for example, indicatingwhether the amount of dust from fragmentation is out of the ordinary.Other outputs can provide other relevant information for the user.

Various types of procedure tasks can be evaluated using the scoringprocess 1000. For example, some tasks can include: number of times auser attempts to insert a needle until the user successfully inserts theneedle; counting time taken to survey a kidney before selecting a targetcalyx for percutaneous access; counting a number of times a navigationalfield generator for tracking a needle is repositioned; counting a numberof times an automated alignment of the robotic manipulator with acatheter is initiated; and counting a number of times a view of thevideo capture device becomes occluded by dust from fragmentation of astone.

Example Robotic System

FIG. 11 illustrates example details of the robotic system 110 inaccordance with one or more embodiments. In this example, the roboticsystem 110 is illustrated as a cart-based robotically-enabled systemthat is movable. However, the robotic system 110 can be implemented as astationary system, integrated into a table, and so on.

The robotic system 110 can include the support structure 114 includingan elongated section 114(A) (sometimes referred to as the “column114(A)”) and a base 114(B). The column 114(A) can include one or morecarriages, such as a carriage 1102 (alternatively referred to as “thearm support 1102”) for supporting the deployment of one or more therobotic arms 112 (three shown in the figure). The carriage 1102 caninclude individually configurable arm mounts that rotate along aperpendicular axis to adjust the base of the robotic arms 112 forpositioning relative to a patient. The carriage 1102 can also include acarriage interface 1104 that allows the carriage 1102 to verticallytranslate along the column 114(A). The carriage interface 1104 isconnected to the column 114(A) through slots, such as slot 1106, thatare positioned on opposite sides of the column 114(A) to guide thevertical translation of the carriage 1102. The slot 1106 includes avertical translation interface to position and hold the carriage 1102 atvarious vertical heights relative to the base 114(B). Verticaltranslation of the carriage 1102 allows the robotic system 110 to adjustthe reach of the robotic arms 112 to meet a variety of table heights,patient sizes, physician preferences. etc. Similarly, the individuallyconfigurable arm mounts on the carriage 1102 allow a robotic arm base1108 of the robotic arms 112 to be angled in a variety ofconfigurations. The column 114(A) can internally comprise mechanisms,such as gears and/or motors, that are designed to use a verticallyaligned lead screw to translate the carriage 1102 in a mechanizedfashion in response to control signals generated in response to userinputs, such as inputs from the I/O device(s) 116.

In some embodiments, the slot 1106 can be supplemented with a slotcover(s) that is flush and/or parallel to the slot surface to preventdirt and/or fluid ingress into the internal chambers of the column114(A) and/or the vertical translation interface as the carriage 1102vertically translates. The slot covers can be deployed through pairs ofspring spools positioned near the vertical top and bottom of the slot1106. The covers can be coiled within the spools until deployed toextend and retract from their coiled state as the carriage 1102vertically translates up and down. The spring-loading of the spools canprovide force to retract the cover into a spool when the carriage 1102translates towards the spool, while also maintaining a tight seal whenthe carriage 1102 translates away from the spool. The covers can beconnected to the carriage 1102 using, for example, brackets in thecarriage interface 1104 to ensure proper extension and retraction of thecovers as the carriage 1102 translates.

The base 114(B) can balance the weight of the column 114(A), thecarriage 1102, and/or arms 112 over a surface, such as the floor.Accordingly, the base 114(B) can house heavier components, such as oneor more electronics, motors, power supply, etc., as well as componentsthat enable movement and/or immobilize the robotic system 110. Forexample, the base 114(B) can include rollable wheels 1116 (also referredto as “the casters 1116”) that allow for the robotic system 110 to movearound the room for a procedure. After reaching an appropriate position,the casters 1116 can be immobilized using wheel locks to hold therobotic system 110 in place during the procedure. As shown, the roboticsystem 110 also includes a handle 1118 to assist with maneuvering and/orstabilizing the robotic system 110.

The robotic arms 112 can generally comprise robotic arm bases 1108 andend effectors 1110, separated by a series of linkages 1112 that areconnected by a series of joints 1114. Each joint 1114 can comprise anindependent actuator and each actuator can comprise an independentlycontrollable motor. Each independently controllable joint 1114represents an independent degree of freedom available to the robotic arm112. For example, each of the arms 112 can have seven joints, and thus,provide seven degrees of freedom. However, any number of joints can beimplemented with any degrees of freedom. In examples, a multitude ofjoints can result in a multitude of degrees of freedom, allowing for“redundant” degrees of freedom. Redundant degrees of freedom allow therobotic arms 112 to position their respective end effectors 1110 at aspecific position, orientation, and/or trajectory in space usingdifferent linkage positions and/or joint angles. In some embodiments,the end effectors 1110 can be configured to engage with and/or control amedical instrument, a device, an object, and so on. The freedom ofmovement of the arms 112 can allow the robotic system 110 to positionand/or direct a medical instrument from a desired point in space and/orallow a physician to move the arms 112 into a clinically advantageousposition away from the patient to create access, while avoiding armcollisions.

As shown in FIG. 11 , the robotic system 110 can also include the I/Odevice(s) 116. The I/O device(s) 116 can include a display, atouchscreen, a touchpad, a projector, a mouse, a keyboard, a microphone,a speaker, a controller, a camera (e.g., to receive gesture input), oranother I/O device to receive input and/or provide output. The I/Odevice(s) 116 can be configured to receive touch, speech, gesture, orany other type of input. The I/O device(s) 116 can be positioned at thevertical end of column 114(A) (e.g., the top of the column 114(A))and/or provide a user interface for receiving user input and/or forproviding output. For example, the I/O device(s) 116 can include atouchscreen (e.g., a dual-purpose device) to receive input and provide aphysician with pre-operative and/or intra-operative data. Examplepre-operative data can include pre-operative plans, navigation, and/ormapping data derived from pre-operative computerized tomography (CT)scans, and/or notes from pre-operative patient interviews. Exampleintra-operative data can include optical information provided from atool/instrument, sensor, and/or coordinate information from sensors, aswell as vital patient statistics, such as respiration, heart rate,and/or pulse. The I/O device(s) 116 can be positioned and/or tilted toallow a physician to access the I/O device(s) 116 from a variety ofpositions, such as the side of the column 114(A) opposite the carriage1102. From this position, the physician can view the I/O device(s) 116,the robotic arms 112, and/or a patient while operating the I/O device(s)116 from behind the robotic system 110.

The robotic system 110 can include a variety of other components. Forexample, the robotic system 110 can include one or more controlelectronics/circuitry, power sources, pneumatics, optical sources,actuators (e.g., motors to move the robotic arms 112), memory, and/orcommunication interfaces (e.g., to communicate with another device). Insome embodiments, the memory can store computer-executable instructionsthat, when executed by the control circuitry, cause the controlcircuitry to perform any of the operations discussed herein. Forexample, the memory can store computer-executable instructions that,when executed by the control circuitry, cause the control circuitry toreceive input and/or a control signal regarding manipulation of therobotic arms 112 and, in response, control the robotic arms 112 to bepositioned in a particular arrangement and/or to navigate a medicalinstrument connected to the end effectors 1110.

In some embodiments, robotic system 110 is configured to engage withand/or control a medical instrument, such as the basket retrieval device120. For example, the robotic arms 112 can be configured to control aposition, orientation, and/or tip articulation of a scope (e.g., asheath and/or a leader of the scope). In some embodiments, the roboticarms 112 can be configured/configurable to manipulate the scope usingelongate movement members. The elongate movement members can include oneor more pull wires (e.g., pull or push wires), cables, fibers, and/orflexible shafts. To illustrate, the robotic arms 112 can be configuredto actuate multiple pull wires coupled to the scope to deflect the tipof the scope. Pull wires can include any suitable or desirablematerials, such as metallic and/or non-metallic materials such asstainless steel, Kevlar, tungsten, carbon fiber, and the like. In someembodiments, the scope is configured to exhibit nonlinear behavior inresponse to forces applied by the elongate movement members. Thenonlinear behavior can be based on stiffness and compressibility of thescope, as well as variability in slack or stiffness between differentelongate movement members.

Example Control System

FIG. 12 illustrates example details of the control system 140 inaccordance with one or more embodiments. As illustrated, the controlsystem 140 can include one or more of the following components, devices,modules, and/or units (referred to herein as “components”), eitherseparately/individually and/or in combination/collectively: controlcircuitry 1202, data storage/memory 1204, one or more communicationinterfaces 1206, one or more power supply units 1208, one or more I/Ocomponents 1210, and/or one or more wheels 1212 (e.g., casters or othertypes of wheels). In some embodiments, the control system 140 cancomprise a housing/enclosure configured and/or dimensioned to house orcontain at least part of one or more of the components of the controlsystem 140. In this example, the control system 140 is illustrated as acart-based system that is movable with the one or more wheels 1212. Insome cases, after reaching the appropriate position, the one or morewheels 1212 can be immobilized using wheel locks to hold the controlsystem 140 in place. However, the control system 140 can be implementedas a stationary system, integrated into another system/device, and soon.

Although certain components of the control system 140 are illustrated inFIG. 12 , it should be understood that additional components not showncan be included in embodiments in accordance with the presentdisclosure. For example, graphical processing units (GPUs) or otherspecialized embedded chips may be included for running neural networks.Furthermore, certain of the illustrated components can be omitted insome embodiments. Although the control circuitry 1202 is illustrated asa separate component in the diagram of FIG. 12 , it should be understoodthat any or all of the remaining components of the control system 140can be embodied at least in part in the control circuitry 1202. That is,the control circuitry 1202 can include various devices (active and/orpassive), semiconductor materials and/or areas, layers, regions, and/orportions thereof, conductors, leads, vias, connections, and/or the like,wherein one or more of the other components of the control system 140and/or portion(s) thereof can be formed and/or embodied at least in partin/by such circuitry components/devices.

The various components of the control system 140 can be electricallyand/or communicatively coupled using certain connectivitycircuitry/devices/features, which can or may not be part of the controlcircuitry 1202. For example, the connectivity feature(s) can include oneor more printed circuit boards configured to facilitate mounting and/orinterconnectivity of at least some of the various components/circuitryof the control system 140. In some embodiments, two or more of thecontrol circuitry 1202, the data storage/memory 1204, the communicationinterface(s) 1206, the power supply unit(s) 1208, and/or theinput/output (I/O) component(s) 1210, can be electrically and/orcommunicatively coupled to each other.

As illustrated, the memory 1204 can include an input device manager 1216and a user interface component 1218 configured to facilitate variousfunctionality discussed herein. In some embodiments, the input devicemanager 1216, and/or the user interface component 1218 can include oneor more instructions that are executable by the control circuitry 1202to perform one or more operations. Although many embodiments arediscussed in the context of the components 1216-1218 including one ormore instructions that are executable by the control circuitry 1202, anyof the components 1216-1218 can be implemented at least in part as oneor more hardware logic components, such as one or more applicationspecific integrated circuits (ASIC), one or more field-programmable gatearrays (FPGAs), one or more program-specific standard products (ASSPs),one or more complex programmable logic devices (CPLDs), and/or the like.Furthermore, although the components 1216-1218 are illustrated as beingincluded within the control system 140, any of the components 1216-1218can be implemented at least in part within another device/system, suchas the robotic system 110, the table 150, or another device/system.Similarly, any of the other components of the control system 140 can beimplemented at least in part within another device/system.

The input device manager 1216 can be configured to receive inputs fromthe input device 146 and translate them into actions performable by therobotic system 110. For example, pre-programmed motions, such as rapidopen, rapid close, and jiggle motion, can be stored in the input devicemanager 1216. These pre-programmed motions can then be assigned to thedesired input (e.g., single or dual button presses, voice commands,joystick movements, etc.). In some implementations, the pre-programmedmotions are determined by the manufacturer. In other implementations,users may be able to modify existing pre-programmed motions and/orcreate new ones.

The user interface component 1218 can be configured to facilitate one ormore user interfaces (also referred to as “one or more graphical userinterfaces (GUI)”). For example, the user interface component 1218 cangenerate a configuration menu for assigning pre-programmed motions toinputs or a settings menu for enabling certain modes of operation ordisabling selected pre-programmed motions in specific situations. Theuser interface component 1218 can also provide user interface data 1222for display to the user.

The one or more communication interfaces 1206 can be configured tocommunicate with one or more device/sensors/systems. For example, theone or more communication interfaces 1206 can send/receive data in awireless and/or wired manner over a network. A network in accordancewith embodiments of the present disclosure can include a local areanetwork (LAN), wide area network (WAN) (e.g., the Internet), personalarea network (PAN), body area network (BAN), etc. In some embodiments,the one or more communication interfaces 1206 can implement a wirelesstechnology such as Bluetooth, Wi-Fi, near field communication (NFC), orthe like.

The one or more power supply units 1208 can be configured to managepower for the control system 140 (and/or the robotic system 110, in somecases). In some embodiments, the one or more power supply units 1208include one or more batteries, such as a lithium-based battery, alead-acid battery, an alkaline battery, and/or another type of battery.That is, the one or more power supply units 1208 can comprise one ormore devices and/or circuitry configured to provide a source of powerand/or provide power management functionality. Moreover, in someembodiments the one or more power supply units 1208 include a mainspower connector that is configured to couple to an alternating current(AC) or direct current (DC) mains power source.

The one or more I/O components 1210 can include a variety of componentsto receive input and/or provide output, such as to interface with auser. The one or more I/O components 1210 can be configured to receivetouch, speech, gesture, or any other type of input. In examples, the oneor more I/O components 1210 can be used to provide input regardingcontrol of a device/system, such as to control the robotic system 110,navigate the scope or other medical instrument attached to the roboticsystem 110, control the table 150, control the fluoroscopy device 190,and so on. As shown, the one or more I/O components 1210 can include theone or more displays 142 (sometimes referred to as “the one or moredisplay devices 142”) configured to display data. The one or moredisplays 142 can include one or more liquid-crystal displays (LCD),light-emitting diode (LED) displays, organic LED displays, plasmadisplays, electronic paper displays, and/or any other type(s) oftechnology. In some embodiments, the one or more displays 142 includeone or more touchscreens configured to receive input and/or displaydata. Further, the one or more I/O components 1210 can include the oneor more input devices 146, which can include a touchscreen, touch pad,controller, mouse, keyboard, wearable device (e.g., optical head-mounteddisplay), virtual or augmented reality device (e.g., head-mounteddisplay), etc. Additionally, the one or more I/O components 1210 caninclude one or more speakers 1226 configured to output sounds based onaudio signals and/or one or more microphones 1228 configured to receivesounds and generate audio signals. In some embodiments, the one or moreI/O components 1210 include or are implemented as a console.

Although not shown in FIG. 9 , the control system 140 can include and/orcontrol other components, such as one or more pumps, flow meters, valvecontrols, and/or fluid access components in order to provide controlledirrigation and/or aspiration capabilities to a medical instrument (e.g.,a scope), a device that can be deployed through a medical instrument,and so on. In some embodiments, irrigation and aspiration capabilitiescan be delivered directly to a medical instrument through separatecable(s). Further, the control system 140 can include a voltage and/orsurge protector designed to provide filtered and/or protected electricalpower to another device, such as the robotic system 110, therebyavoiding placement of a power transformer and other auxiliary powercomponents in the robotic system 110, resulting in a smaller, moremoveable robotic system 110.

The control system 140 can also include support equipment for sensorsdeployed throughout the medical system 100. For example, the controlsystem 140 can include opto-electronics equipment for detecting,receiving, and/or processing data received from optical sensors and/orcameras. Such opto-electronics equipment can be used to generatereal-time images for display in any number of devices/systems, includingin the control system 140.

In some embodiments, the control system 140 can be coupled to therobotic system 110, the table 150, and/or a medical instrument, such asthe scope and/or the basket retrieval device 120, through one or morecables or connections (not shown). In some implementations, supportfunctionality from the control system 140 can be provided through asingle cable, simplifying and de-cluttering an operating room. In otherimplementations, specific functionality can be coupled in separatecabling and connections. For example, while power can be providedthrough a single power cable, the support for controls, optics,fluidics, and/or navigation can be provided through a separate cable.

The term “control circuitry” is used herein according to its broad andordinary meaning, and can refer to any collection of one or moreprocessors, processing circuitry, processing modules/units, chips, dies(e.g., semiconductor dies including come or more active and/or passivedevices and/or connectivity circuitry), microprocessors,micro-controllers, digital signal processors, microcomputers, centralprocessing units, graphics processing units, field programmable gatearrays, programmable logic devices, state machines (e.g., hardware statemachines), logic circuitry, analog circuitry, digital circuitry, and/orany device that manipulates signals (analog and/or digital) based onhard coding of the circuitry and/or operational instructions. Controlcircuitry can further comprise one or more, storage devices, which canbe embodied in a single memory device, a plurality of memory devices,and/or embedded circuitry of a device. Such data storage can compriseread-only memory, random access memory, volatile memory, non-volatilememory, static memory, dynamic memory, flash memory, cache memory, datastorage registers, and/or any device that stores digital information. Itshould be noted that in embodiments in which control circuitry comprisesa hardware state machine (and/or implements a software state machine),analog circuitry, digital circuitry, and/or logic circuitry, datastorage device(s)/register(s) storing any associated operationalinstructions can be embedded within, or external to, the circuitrycomprising the state machine, analog circuitry, digital circuitry,and/or logic circuitry.

The term “memory” is used herein according to its broad and ordinarymeaning and can refer to any suitable or desirable type ofcomputer-readable media. For example, computer-readable media caninclude one or more volatile data storage devices, non-volatile datastorage devices, removable data storage devices, and/or nonremovabledata storage devices implemented using any technology, layout, and/ordata structure(s)/protocol, including any suitable or desirablecomputer-readable instructions, data structures, program modules, orother types of data.

Computer-readable media that can be implemented in accordance withembodiments of the present disclosure includes, but is not limited to,phase change memory, static random-access memory (SRAM), dynamicrandom-access memory (DRAM), other types of random access memory (RAM),read-only memory (ROM), electrically erasable programmable read-onlymemory (EEPROM), flash memory or other memory technology, compact diskread-only memory (CD-ROM), digital versatile disks (DVD) or otheroptical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices, or any other non-transitorymedium that can be used to store information for access by a computingdevice. As used in certain contexts herein, computer-readable media maynot generally include communication media, such as modulated datasignals and carrier waves. As such, computer-readable media shouldgenerally be understood to refer to non-transitory media.

Additional Embodiments

Depending on the embodiment, certain acts, events, or functions of anyof the processes or algorithms described herein can be performed in adifferent sequence, may be added, merged, or left out altogether. Thus,in certain embodiments, not all described acts or events are necessaryfor the practice of the processes.

Conditional language used herein, such as, among others, “can,” “could,”“might,” “may,” “e.g.,” and the like, unless specifically statedotherwise, or otherwise understood within the context as used, isintended in its ordinary sense and is generally intended to convey thatcertain embodiments include, while other embodiments do not include,certain features, elements and/or steps. Thus, such conditional languageis not generally intended to imply that features, elements and/or stepsare in any way required for one or more embodiments or that one or moreembodiments necessarily include logic for deciding, with or withoutauthor input or prompting, whether these features, elements and/or stepsare included or are to be performed in any particular embodiment. Theterms “comprising,” “including,” “having,” and the like are synonymous,are used in their ordinary sense, and are used inclusively, in anopen-ended fashion, and do not exclude additional elements, features,acts, operations, and so forth. Also, the term “or” is used in itsinclusive sense (and not in its exclusive sense) so that when used, forexample, to connect a list of elements, the term “or” means one, some,or all of the elements in the list. Conjunctive language such as thephrase “at least one of X, Y, and Z,” unless specifically statedotherwise, is understood with the context as used in general to conveythat an item, term, element, etc. may be either X, Y, or Z. Thus, suchconjunctive language is not generally intended to imply that certainembodiments require at least one of X, at least one of Y, and at leastone of Z to each be present.

It should be appreciated that in the above description of embodiments,various features are sometimes grouped together in a single embodiment,Figure, or description thereof for the purpose of streamlining thedisclosure and aiding in the understanding of one or more of the variousinventive aspects. This method of disclosure, however, is not to beinterpreted as reflecting an intention that any claim require morefeatures than are expressly recited in that claim. Moreover, anycomponents, features, or steps illustrated and/or described in aparticular embodiment herein can be applied to or used with any otherembodiment(s). Further, no component, feature, step, or group ofcomponents, features, or steps are necessary or indispensable for eachembodiment. Thus, it is intended that the scope of the inventions hereindisclosed and claimed below should not be limited by the particularembodiments described above, but should be determined only by a fairreading of the claims that follow.

It should be understood that certain ordinal terms (e.g., “first” or“second”) may be provided for ease of reference and do not necessarilyimply physical characteristics or ordering. Therefore, as used herein,an ordinal term (e.g., “first,” “second,” “third,” etc.) used to modifyan element, such as a structure, a component, an operation, etc., doesnot necessarily indicate priority or order of the element with respectto any other element, but rather may generally distinguish the elementfrom another element having a similar or identical name (but for use ofthe ordinal term). In addition, as used herein, indefinite articles (“a”and “an”) may indicate “one or more” rather than “one.” Further, anoperation performed “based on” a condition or event may also beperformed based on one or more other conditions or events not explicitlyrecited.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which example embodiments belong. It befurther understood that terms, such as those defined in commonly useddictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art and notbe interpreted in an idealized or overly formal sense unless expresslyso defined herein.

Unless otherwise expressly stated, comparative and/or quantitativeterms, such as “less,” “more,” “greater,” and the like, are intended toencompass the concepts of equality. For example, “less” can mean notonly “less” in the strictest mathematical sense, but also, “less than orequal to.”

What is claimed is:
 1. A robotic system for evaluating an identifiedphase of a medical procedure performed by the robotic system, therobotic system comprising: a video capture device; a roboticmanipulator; one or more sensors configured to detect a configuration ofthe robotic manipulator; an input device configured to receive one ormore user interactions and initiate one or more actions by the roboticmanipulator; a data store configured to store metrics associated withphases of medical procedures; and control circuitry communicativelycoupled to the input device and robotic manipulator, the controlcircuitry configured to: determine a first status of the roboticmanipulator based on sensor data from the one or more sensors; identifya first input from the input device for initiating a first action of therobotic manipulator; perform a first analysis of a video of a patientsite captured by the video capture device; identify a first phase of themedical procedure based at least in part on the first status of therobotic manipulator, the first input, and the first analysis of thevideo; and generate an evaluation of the first phase of the medicalprocedure based on one or more metrics associated with the first phase.2. The robotic system of claim 1, wherein the first phase comprises oneof ureteroscopy driving, ureteroscopy lasing, ureteroscopy basketing,and percutaneous needle insertion.
 3. The robotic system of claim 1,wherein the first phase comprises ureteroscopy basketing and generatingthe evaluation comprises: counting a number of basket operations;counting a number of ureteroscope retractions; determining a ratio ofthe number of basket operations to the number of ureteroscoperetractions; and comparing the determined ratio with other ratios fromprevious ureteroscopy basketing procedures.
 4. The robotic system ofclaim 1, wherein the first phase comprises ureteroscopy driving andgenerating the evaluation comprises: counting a number of times a userdrives a scope manually; counting a number of times the user drives thescope robotically; determining a ratio of the number of times the userdrives a scope manually to the number of times the user drives the scoperobotically; and comparing the determined ratio with other ratios fromprevious ureteroscopy basketing procedures.
 5. The robotic system ofclaim 1, wherein the first phase comprises percutaneous needle insertionand generating the evaluation comprises: counting a number of times auser attempts to insert a needle until the user successfully inserts theneedle; and comparing the counted number of times with recorded needleinsertion attempts from previous percutaneous needle insertionoperations.
 6. The robotic system of claim 1, wherein the first phasecomprises percutaneous needle insertion and generating the evaluationcomprises: counting time taken to survey a kidney before selecting atarget calyx for percutaneous access; and comparing the counted timewith recorded times from previous percutaneous needle insertionoperations.
 7. The robotic system of claim 1, wherein the first phasecomprises percutaneous needle insertion and generating the evaluationcomprises: counting a number of times a navigational field generator fortracking a needle is repositioned; and comparing the counted number oftimes with recorded repositioning numbers from previous percutaneousneedle insertion operations.
 8. The robotic system of claim 1, whereinthe first phase comprises percutaneous needle insertion and generatingthe evaluation comprises: counting a number of times an automatedalignment of an end effector of the robotic manipulator with a catheteris initiated; and comparing the counted number of times with recordedautomated alignment numbers from previous operations.
 9. The roboticsystem of claim 1, wherein the first phase comprises ureteroscopy lasingand generating the evaluation comprises: counting a lasing time for astone; determining a size of the stone; and comparing a ratio of thelasing time to the size of the stone with previous ratios from otheroperations.
 10. The robotic system of claim 1, wherein the first phasecomprises ureteroscopy lasing and generating the evaluation comprises:determining a type of a stone; and aggregating statistics acrosssurgical operations based on the type of the stone.
 11. The roboticsystem of claim 1, wherein the first phase comprises ureteroscopy lasingand generating the evaluation comprises: counting a number of times aview of the video capture device becomes occluded by dust fromfragmentation of a stone; and comparing the counted number of times withrecorded number of dust occlusions from operations.
 12. A method forevaluating an identified phase of a medical procedure performed by arobotic system comprising a video capture device, a robotic manipulator,one or more sensors, and an input device, the method comprising:determining a first status of the robotic manipulator based on sensordata from the one or more sensors; identifying a first input from theinput device for initiating a first action of the robotic manipulator;performing a first analysis of a video of a patient site captured by thevideo capture device; identifying a first phase of the medical procedurebased at least in part on the first status of the robotic manipulator,the first input, and the first analysis of the video; and generating anevaluation of the first phase of the medical procedure based on one ormore metrics associated with the first phase.
 13. The method of claim12, wherein the first phase comprises ureteroscopy basketing andgenerating the evaluation comprises: counting a number of basketoperations; counting a number of ureteroscope retractions; determining aratio of the number of basket operations to the number of ureteroscoperetractions; and comparing the determined ratio with other ratios fromprevious ureteroscopy basketing operations.
 14. The method of claim 12,wherein the first phase comprises ureteroscopy driving and generatingthe evaluation comprises: counting a number of times a user drives ascope manually; counting a number of times the user drives the scoperobotically; determining a ratio of the number of times the user drivesa scope manually to the number of times the user drives the scoperobotically; and comparing the determined ratio with other ratios fromprevious ureteroscopy basketing operations.
 15. The method of claim 12,wherein the first phase comprises percutaneous needle insertion andgenerating the evaluation comprises: counting a number of times a userattempts to insert a needle until the user successfully inserts theneedle; and comparing the counted number of times with recorded needleinsertion attempts from previous percutaneous needle insertionoperations.
 16. The method of claim 12, wherein the first phasecomprises percutaneous needle insertion and generating the evaluationcomprises: counting time taken to survey a kidney before selecting atarget calyx for percutaneous access; and comparing the counted timewith recorded times from previous percutaneous needle insertionoperations.
 17. The method of claim 12, wherein the first phasecomprises percutaneous needle insertion and generating the evaluationcomprises: counting a number of times a navigational field generator fortracking a needle is repositioned; and comparing the counted number oftimes with recorded repositioning numbers from previous percutaneousneedle insertion operations.
 18. The method of claim 12, wherein thefirst phase comprises percutaneous needle insertion and generating theevaluation comprises: counting a number of times an automated alignmentof an end effector of the robotic manipulator with a catheter isinitiated; and comparing the counted number of times with recordedautomated alignment numbers from previous operations.
 19. The method ofclaim 12, wherein the first phase comprises percutaneous antegradeureteroscopy lasing and generating the evaluation comprises: counting alasing time for a stone; determining a size of the stone; and comparinga ratio of the lasing time to the size of the stone with previous ratiosfrom other operations.
 20. A control system of a robotic device forevaluating an identified phase of a medical procedure, the controlsystem comprising: a communication interface configured to receivesensor data, user input data, and video data from the robotic device;memory configured to store the sensor data, the user input data, and thevideo data; and one or more processors configured to: determine a firststatus of a manipulator of the robotic device based on sensor data;identify a first input, from the user input data, for initiating a firstaction of the manipulator; perform a first analysis of a video of apatient site captured by a video capture device; identify a first phaseof the medical procedure based at least in part on the first status ofthe manipulator, the first input, and the first analysis of the video;and generate an evaluation of the first phase of the medical procedurebased on one or more metrics associated with the first phase.